Evolutionary Reinforcement Learning Research Articles

Multi-objective reinforcement learning based on nonlinear scalarization and long-short-term optimization

Purpose Many practical control problems require achieving multiple objectives, and these objectives often conflict with each other. The existing multi-objective evolutionary reinforcement learning algorithms cannot achieve good search results when solving such problems. It is necessary to design a new multi-objective evolutionary reinforcement learning algorithm with a stronger searchability. Design/methodology/approach The multi-objective reinforcement learning algorithm proposed in this paper is based on the evolutionary computation framework. In each generation, this study uses the long-short-term selection method to select parent policies. The long-term selection is based on the improvement of policy along the predefined optimization direction in the previous generation. The short-term selection uses a prediction model to predict the optimization direction that may have the greatest improvement on overall population performance. In the evolutionary stage, the penalty-based nonlinear scalarization method is used to scalarize the multi-dimensional advantage functions, and the nonlinear multi-objective policy gradient is designed to optimize the parent policies along the predefined directions. Findings The penalty-based nonlinear scalarization method can force policies to improve along the predefined optimization directions. The long-short-term optimization method can alleviate the exploration-exploitation problem, enabling the algorithm to explore unknown regions while ensuring that potential policies are fully optimized. The combination of these designs can effectively improve the performance of the final population. Originality/value A multi-objective evolutionary reinforcement learning algorithm with stronger searchability has been proposed. This algorithm can find a Pareto policy set with better convergence, diversity and density.

Advancements in Reinforcement Learning Algorithms for Autonomous Systems

Reinforcement learning, often known as RL, has developed as a strong paradigm to teach autonomous software agents to make choices in contexts that are both complicated and dynamic. This abstract investigates recent developments and uses of RL in a variety of fields, showing both its transformational potential and the constraints that it faces at present. Recent developments in reinforcement learning (RL) algorithms, in particular deep reinforcement learning (DRL), have made it possible to make major advancements in autonomous decision- making tasks. DRL algorithms can learn complicated representations of state-action spaces by using deep neural networks. This allows for more efficient exploration and exploitation methods to be implemented. Additionally, advancements in algorithmic enhancements, such as prioritized experience replay and distributional reinforcement learning, have improved the stability and sample efficiency of reinforcement learning algorithms, which has made it possible for these algorithms to be used in real-world applications. Robotics, autonomous cars, game playing, finance, and healthcare are just a few of the many fields that may benefit from the use of RL. In the field of robotics, reinforcement learning (RL) makes it possible for autonomous agents to learn how to navigate, manipulate, and move about in environments that are both complicated and unstructured. To improve both safety and efficiency on the road, autonomous cars make use of reinforcement learning (RL) to make decisions in dynamic traffic situations. In finance, RL algorithms are used for portfolio optimization, algorithmic trading, and risk management. These applications serve to improve investment techniques and decision-making procedures. Furthermore, in the field of healthcare, RL supports individualized treatment planning, clinical decision support, and medical image analysis, which enables physicians to provide patients with care that is specifically suited to their needs. Despite the promising improvements and applications, RL is still confronted with several difficulties that restrict its capacity to be widely adopted and scaled. Among these problems are the inefficiency of the sample, the trade-offs between exploration and exploitation, concerns about safety and dependability, and the need for explainability and interpretability in decision-making processes. To effectively address these difficulties, it is necessary to engage in collaborative efforts across several disciplines, conduct research on algorithmic developments, and establish extensive assessment frameworks (Anon, 2022).

A Multiagent Cooperative Learning System With Evolution of Social Roles

Recent developments in reinforcement learning have been able to derive optimal policies for sophisticated and capable agents, and shown to achieve human-level performance on a number of challenging tasks. Unfortunately, when it comes to multi-agent systems (MASs), complexities such as non-stationarity and partial observability bring new challenges to the field. Building a flexible and efficient multi-agent reinforcement learning algorithm capable of handling complex tasks has to date remained an open challenge. This paper presents an Multi-Agent learning system with the evolution of Social Roles (eSRMA). The main interest is placed on solving the key issues in the definition and evolution of suitable roles, and optimizing the policies accompanied by social roles in MAS efficiently. Specifically, eSRMA incorporates and cultivates role division awareness of agents to improve the ability to deal with complex cooperative tasks. Each agent is assigned a role module, which can dynamically generate roles based on the individuals’ local observations. A novel multi-agent reinforcement learning algorithm (MARL) is designed as the principal driving force that governs the role-policy learning process by a role-attention credit assignment mechanism. Moreover, a role evolution process is developed to help agents dynamically choose appropriate roles in decision-making. Comprehensive experiments on the StarCraft II micromanagement benchmark have demonstrated that eSRMA exhibits superiority in achieving higher learning capability and efficiency for multiple agents compared to the state-of-the-art MARL methods.

ERL-TD: Evolutionary Reinforcement Learning Enhanced with Truncated Variance and Distillation Mutation

Recently, an emerging research direction called Evolutionary Reinforcement Learning (ERL) has been proposed, which combines evolutionary algorithm with reinforcement learning (RL) for tackling the tasks of sequential decision making. However, the recently proposed ERL algorithms often suffer from two challenges: the inaccuracy of policy estimation caused by the overestimation bias in RL and the insufficiency of exploration caused by inefficient mutations. To alleviate these problems, we propose an Evolutionary Reinforcement Learning algorithm enhanced with Truncated variance and Distillation mutation, called ERL-TD. We utilize multiple Q-networks to evaluate state-action pairs, so that multiple networks can provide more accurate evaluations for state-action pairs, in which the variance of evaluations can be adopted to control the overestimation bias in RL. Moreover, we propose a new distillation mutation to provide a promising mutation direction, which is different from traditional mutation generating a large number of random solutions. We evaluate ERL-TD on the continuous control benchmarks from the OpenAI Gym and DeepMind Control Suite. The experiments show that ERL-TD shows excellent performance and outperforms all baseline RL algorithms on the test suites.

Scale Optimization Using Evolutionary Reinforcement Learning for Object Detection on Drone Imagery

Object detection in aerial imagery presents a significant challenge due to large scale variations among objects. This paper proposes an evolutionary reinforcement learning agent, integrated within a coarse-to-fine object detection framework, to optimize the scale for more effective detection of objects in such images. Specifically, a set of patches potentially containing objects are first generated. A set of rewards measuring the localization accuracy, the accuracy of predicted labels, and the scale consistency among nearby patches are designed in the agent to guide the scale optimization. The proposed scale-consistency reward ensures similar scales for neighboring objects of the same category. Furthermore, a spatial-semantic attention mechanism is designed to exploit the spatial semantic relations between patches. The agent employs the proximal policy optimization strategy in conjunction with the evolutionary strategy, effectively utilizing both the current patch status and historical experience embedded in the agent. The proposed model is compared with state-of-the-art methods on two benchmark datasets for object detection on drone imagery. It significantly outperforms all the compared methods. Code is available at https://github.com/UNNC-CV/EvOD/.

Designing Biological Sequences without Prior Knowledge Using Evolutionary Reinforcement Learning

Designing novel biological sequences with desired properties is a significant challenge in biological science because of the extra large search space. The traditional design process usually involves multiple rounds of costly wet lab evaluations. To reduce the need for expensive wet lab experiments, machine learning methods are used to aid in designing biological sequences. However, the limited availability of biological sequences with known properties hinders the training of machine learning models, significantly restricting their applicability and performance. To fill this gap, we present ERLBioSeq, an Evolutionary Reinforcement Learning algorithm for BIOlogical SEQuence design. ERLBioSeq leverages the capability of reinforcement learning to learn without prior knowledge and the potential of evolutionary algorithms to enhance the exploration of reinforcement learning in the large search space of biological sequences. Additionally, to enhance the efficiency of biological sequence design, we developed a predictor for sequence screening in the biological sequence design process, which incorporates both the local and global sequence information. We evaluated the proposed method on three main types of biological sequence design tasks, including the design of DNA, RNA, and protein. The results demonstrate that the proposed method achieves significant improvement compared to the existing state-of-the-art methods.

Two-Stage Evolutionary Reinforcement Learning for Enhancing Exploration and Exploitation

The integration of Evolutionary Algorithm (EA) and Reinforcement Learning (RL) has emerged as a promising approach for tackling some challenges in RL, such as sparse rewards, lack of exploration, and brittle convergence properties. However, existing methods often employ actor networks as individuals of EA, which may constrain their exploratory capabilities, as the entire actor population will stop evolution when the critic network in RL falls into local optimal. To alleviate this issue, this paper introduces a Two-stage Evolutionary Reinforcement Learning (TERL) framework that maintains a population containing both actor and critic networks. TERL divides the learning process into two stages. In the initial stage, individuals independently learn actor-critic networks, which are optimized alternatively by RL and Particle Swarm Optimization (PSO). This dual optimization fosters greater exploration, curbing susceptibility to local optima. Shared information from a common replay buffer and PSO algorithm substantially mitigates the computational load of training multiple agents. In the subsequent stage, TERL shifts to a refined exploitation phase. Here, only the best individual undergoes further refinement, while the rest individuals continue PSO-based optimization. This allocates more computational resources to the best individual for yielding superior performance. Empirical assessments, conducted across a range of continuous control problems, validate the efficacy of the proposed TERL paradigm.

Lamarckian Platform: Pushing the Boundaries of Evolutionary Reinforcement Learning Toward Asynchronous Commercial Games

Despite the emerging progress of integrating evolutionary computation into reinforcement learning, the absence of a high-performance platform endowing composability and massive parallelism causes non-trivial difficulties for research and applications related to asynchronous commercial games. Here we introduce Lamarckian <xref ref-type="fn" rid="fn1" xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">¹</xref> <fn id="fn1" xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"><label>¹</label> The code and demonstrational setup of Lamarckian are publicly available at <uri>https://github. com/lamarckian/lamarckian</uri>. </fn> – an open-source platform featuring support for evolutionary reinforcement learning scalable to distributed computing resources. To improve the training speed and data efficiency, Lamarckian adopts optimized communication methods and an asynchronous evolutionary reinforcement learning workflow. To meet the demand for an asynchronous interface by commercial games and various methods, Lamarckian tailors an asynchronous Markov Decision Process interface and designs an object-oriented software architecture with decoupled modules. In comparison with the state-of-the-art RLlib, we empirically demonstrate the unique advantages of Lamarckian on benchmark tests with up to 6000 CPU cores: i) both the sampling efficiency and training speed are doubled when running PPO on Google football game; ii) the training speed is 13 times faster when running PBT+PPO on Pong game. Moreover, we also present two use cases: i) how Lamarckian is applied to generating behavior-diverse game AI; ii) how Lamarckian is applied to game balancing tests for an asynchronous commercial game.

Adaptive Evolutionary Reinforcement Learning with Policy Direction

Evolutionary Reinforcement Learning (ERL) has garnered widespread attention in recent years due to its inherent robustness and parallelism. However, the integration of Evolutionary Algorithms (EAs) and Reinforcement Learning (RL) remains relatively rudimentary and lacks dynamism, which can impact the convergence performance of ERL algorithms. In this study, a dynamic adaptive module is introduced to balance the Evolution Strategies (ES) and RL training within ERL. By incorporating elite strategies, this module leverages advantageous individuals to elevate the overall population's performance. Additionally, RL strategy updates often lack guidance from the population. To address this, we incorporate the strategies of the best individuals from the population, providing valuable policy direction. This is achieved through the formulation of a loss function that employs either L1 or L2 regularization to facilitate RL training. The proposed framework is referred to as Adaptive Evolutionary Reinforcement Learning (AERL). The effectiveness of our framework is evaluated by adopting Soft Actor-Critic (SAC) as the RL algorithm and comparing it with other algorithms in the MuJoCo environment. The results underscore the outstanding convergence performance of our proposed Adaptive Evolutionary Soft Actor-Critic (AESAC) algorithm. Furthermore, ablation experiments are conducted to emphasize the necessity of these two improvements. It is worth noting that the enhancements in AESAC are realized at the population level, enabling broader exploration and effectively reducing the risk of falling into local optima.

A modified evolutionary reinforcement learning for multi-agent region protection with fewer defenders

AbstractAutonomous region protection is a significant research area in multi-agent systems, aiming to empower defenders in preventing intruders from accessing specific regions. This paper presents a Multi-agent Region Protection Environment (MRPE) featuring fewer defenders, defender damages, and intruder evasion strategies targeting defenders. MRPE poses challenges for traditional protection methods due to its high nonstationarity and limited interception time window. To surmount these hurdles, we modify evolutionary reinforcement learning, giving rise to the corresponding multi-agent region protection method (MRPM). MRPM amalgamates the merits of evolutionary algorithms and deep reinforcement learning, specifically leveraging Differential Evolution (DE) and Multi-Agent Deep Deterministic Policy Gradient (MADDPG). DE facilitates diverse sample exploration and overcomes sparse rewards, while MADDPG trains defenders and expedites the DE convergence process. Additionally, an elite selection strategy tailored for multi-agent systems is devised to enhance defender collaboration. The paper also presents ingenious designs for the fitness and reward functions to effectively drive policy optimizations. Finally, extensive numerical simulations are conducted to validate the effectiveness of MRPM.

Evolutionary Reinforcement Learning: Hybrid Approach for Safety-Informed Fault-Tolerant Flight Control

Recent research in artificial intelligence potentially provides solutions to the challenging problem of fault-tolerant and robust flight control. This paper proposes a novel Safety-Informed Evolutionary Reinforcement Learning algorithm (SERL), which combines Deep Reinforcement Learning (DRL) and neuroevolution to optimize a population of nonlinear control policies. Using SERL, the work has trained agents to provide attitude tracking on a high-fidelity nonlinear fixed-wing aircraft model. Compared to a state-of-the-art DRL solution, SERL achieves better tracking performance in nine out of ten cases, remaining robust against faults and changes in flight conditions, while providing smoother action signals.

Reducing idleness in financial cloud services via multi-objective evolutionary reinforcement learning based load balancer

Reducing idleness in financial cloud services via multi-objective evolutionary reinforcement learning based load balancer

State-Dependent Maximum Entropy Reinforcement Learning for Robot Long-Horizon Task Learning

Task-oriented robot learning has shown significant potential with the development of Reinforcement Learning (RL) algorithms. However, the learning of long-horizon tasks for robots remains a formidable challenge due to the inherent complexity of tasks, typically comprising multiple diverse stages. Universal RL algorithms commonly encounter issues such as slow convergence or even failure to converge altogether when applied to such tasks. The reasons behind these challenges lie in the local optima trap or redundant exploration during the new stages or the junction of two continuous stages. To address these challenges, we propose a novel state-dependent maximum entropy (SDME) reinforcement learning algorithm. This algorithm effectively balances the trade-off between exploration and exploitation around three kinds of critical states arising from the unique nature of long-horizon tasks. We conducted experiments within an open-source simulation environment, focusing on two representative long-horizon tasks. The proposed SDME algorithm exhibits faster and more stable learning capabilities, requiring merely one-third of the number of learning samples necessary for baseline approaches. Furthermore, we assess the generalization ability of our method under randomly initialized conditions, and the results show that the success rate of the SDME algorithm is nearly twice that of the baselines. Our code will be available at https://github.com/Peter-zds/SDME.

Learning-Based Adaptive Optimal Control of Linear Time-Delay Systems: A Policy Iteration Approach

This paper studies the adaptive optimal control problem for a class of linear time-delay systems described by delay differential equations (DDEs). A crucial strategy is to take advantage of recent developments in reinforcement learning (RL) and adaptive dynamic programming (ADP) and develop novel methods to learn adaptive optimal controllers from finite samples of input and state data. In this paper, the data-driven policy iteration (PI) is proposed to solve the infinite-dimensional algebraic Riccati equation (ARE) iteratively in the absence of exact model knowledge. Interestingly, the proposed recursive PI algorithm is new in the present context of continuous-time time-delay systems, even when the model knowledge is assumed known. The efficacy of the proposed learning-based control methods is validated by means of practical applications arising from metal cutting and autonomous driving.

Editorial to the “Evolutionary Reinforcement Learning” Special Issue

Editorial to the “Evolutionary Reinforcement Learning” Special Issue

Solving an energy resource management problem with a novel multi-objective evolutionary reinforcement learning method

Microgrids have become popular candidates for integrating diverse energy sources into the power grid as means of reducing fossil fuel usage. Energy Resource Management (ERM) is a type of Unit Commitment problem, where a player operates a microgrid with diverse renewable generators integrated with an external supplier. Calculating the economic dispatch of each committed unit on a planning horizon is an NP-hard problem, and therefore, finding an exact solution is difficult. This paper presents a multi-objective solution to the ERM problem from the perspective of battery operation and external supplier dispatch. First, a novel multi-objective decision problem modeling is proposed that considers three objectives: cost, greenhouse gas emissions, and battery degradation. This framework involves a learning agent that controls the depth of discharge of a Lithium-Ion battery. To address the proposed problem, a new multi-objective algorithm called Multi-Objective Evolutionary Policy Search (MEPS) is introduced. The proposed algorithm uses NeuroEvolution of Augmenting Topologies structure to evolve artificial neural networks for estimating action-preference values considering multi-objective rewards. The MEPS performance is evaluated on both standard and newly-proposed benchmark problems, using the hypervolume as the evaluation metric. When compared to standard deep reinforcement learning, results showed that MEPS provides cost-effective, environmentally friendly, and efficient energy storage management solutions. Furthermore, MEPS effectively solves the proposed ERM problem by finding neural networks with a small number of nodes and connections, which are suitable for use in embedded control systems. Overall, MEPS proved to be a promising multi-objective approach in the transition to clean energy resources.

A survey on Evolutionary Reinforcement Learning algorithms

Reinforcement Learning (RL) has proven to be highly effective in various real-world applications. However, in certain scenarios, Evolutionary Algorithms (EAs) have been utilized as an alternative to RL algorithms. Recently, Evolutionary Reinforcement Learning algorithms (ERLs) have emerged as a promising solution that combines the advantages of both RL and EA. This paper presents a comprehensive survey that encompasses a majority of the studies in this exciting research area. We classify these ERLs according to the EA used in their frameworks and analyze the strengths and limitations of various EA components and combination schemes. Additionally, we conduct several experiments to evaluate the performance of some representative ERLs. By categorizing the different approaches and assessing their effectiveness, the paper can assist researchers and practitioners in selecting the most suitable method for their particular application.

Evolutionary reinforcement learning algorithm for large-scale multi-agent cooperation and confrontation applications

Evolutionary reinforcement learning algorithm for large-scale multi-agent cooperation and confrontation applications

Explainable multi-agent deep reinforcement learning for real-time demand response towards sustainable manufacturing

The demand response (DR) plays a significant role in manufacturing system energy management and sustainable industrial development. Current literature on DR management for manufacturing systems has mostly focused on day-ahead production scheduling, whose effectiveness is limited due to the lack of flexibility to control the production line in real time. The development of reinforcement learning (RL) possesses huge potential for real-time production control to address the flexibility issue. However, since production is the top priority for any manufacturing system, a trustful and explainable RL for manufacturing system energy management that can ensure the production requirements is necessary for this application. This study proposes an explainable multi-agent deep RL method, where the analytical manufacturing system model is applied to decompose the system-level energy management objective and production requirement to the agent level. Based on the decomposed task, the agent can then form a safe action subset that is interpretable to achieve the original system-level production requirement while learning to reduce energy costs under DR. The proposed RL method, which is referred to as decomposed multi-agent deep Q-network (DMADQN), is applied to control a section of an automotive assembly line using one year of DR electricity price data to validate its performance. Results show that the proposed method ensures the achievement of the production requirement while providing better DR energy management performance in both RL training and testing phases. In addition, the proposed approach can outperform the day-ahead scheduling approach and save up to an additional 30.7% of energy costs under dynamic DR.

Multi-agent evolution reinforcement learning method for machining parameters optimization based on bootstrap aggregating graph attention network simulated environment

Improving machining quality and production efficiency is the focus of the manufacturing industry. How to obtain efficient machining parameters under multiple constraints such as machining quality is a severe challenge for manufacturing industry. In this paper, a multi-agent evolutionary reinforcement learning method (MAERL) is proposed to optimize the machining parameters for high quality and high efficiency machining by combining the graph neural network and reinforcement learning. Firstly, a bootstrap aggregating graph attention network (Bagging-GAT) based roughness estimation method for machined surface is proposed, which combines the structural knowledge between machining parameters and vibration features. Secondly, a mathematical model of machining parameters optimization problem is established, which is formalized into Markov decision process (MDP), and a multi-agent reinforcement learning method is proposed to solve the MDP problem, and evolutionary learning is introduced to improve the stability of multi-agent training. Finally, a series of experiments were carried out on the commutator production line, and the results show that the proposed Bagging-GAT-based method can improve the prediction effect by about 25% in the case of small samples, and the MAERL-based optimization method can better deal with the coupling problem of reward function in the optimization process. Compared with the classical optimization method, the optimization effect is improved by 13% and a lot of optimization time is saved.